image_types_sparse: use ext2simg_android for ext* sparse images
v3 cover image_types_sparse: use ext2simg_android for ext* sparse images
This series switches ext* sparse image generation from the generic img2simg to ext2simg_android from e2fsprogs contrib/android. Unlike img2simg,which scans the raw image for zero-filled blocks, ext2simg_android understands the ext* filesystem structure and uses the block bitmap to determine sparse areas, which is faster and produces smaller sparse images.
This reduces flashing time for Android RFS/fastboot images and cuts cost in CI/CD pipelines. img2simg is retained for non-ext types such as f2fs.
1. e2fsprogs-ext4sparse-native: a standalone native recipe that compiles ext2simg.c against android-tools' libsparse.
2. android-tools: export libsparse/libbase/liblog headers and libs to the sysroot (under ${includedir}/sparse) so the recipe above can link against them.
3. image_types_sparse.bbclass: branch CONVERSION_CMD:sparse by filesystem type, calling ext2simg_android for ext* and falling back to img2simg otherwise.
Changes in v3:
- Android tools has been moved away from selinux dynamic-layers
- Reworked into a standalone e2fsprogs-ext4sparse recipe in meta-oe as per feedback on the RFC
- Included the previously-missing e2fsprogs-ext4sparse recipe file that was absent from the v2 (RFC_V2) attachment.
Changes in v2 (RFC_V2):
- Split the single cross-layer PoC into a dedicated ext2simg_android recipe rather than patching e2fsprogs-native directly.
RFC (v1):
- Initial cross-layer proof of concept using a manual ${CC} call in do_compile:append and LD_LIBRARY_PATH for runtime linking.
AshishKumar Mishra (3):
e2fsprogs-ext4sparse-native: compiles ext2simg.c using android-tools-native
android-tools: export libsparse/libbase/liblog headers and libs to sysroot
image_types_sparse: switch ext* conversion to ext2simg_android
meta-oe/classes/image_types_sparse.bbclass | 16 ++++-- .../android-tools/android-tools_%.bbappend | 54 +++++++++++++++++++ .../e2fsprogs-ext4sparse-native_1.47.4.bb | 34 ++++++++++++ 3 files changed, 101 insertions(+), 3 deletions(-) create mode 100644 meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend create mode 100644 meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb
Signature · 1 line
2.43.0
PATCH v3 1/3 e2fsprogs-ext4sparse-native: compiles ext2simg.c using android-tools-native
Add a custom do_compile step to build the ext2simg_android utility (ext2simg) when building for the native class.
This can be used to generate ext* based sparse image
--- .../e2fsprogs-ext4sparse-native_1.47.4.bb | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb
meta-oe/recipes-devtools/e2fsprogs-ext4sparse/e2fsprogs-ext4sparse-native_1.47.4.bb new file +32 -0
@@ -0,0 +1,32 @@ +SUMMARY = "Native ext sparse conversion utility from e2fsprogs" +DESCRIPTION = "Builds only ext2simg for host-side ext* sparse image conversion" +HOMEPAGE = "http://e2fsprogs.sourceforge.net/" +LICENSE = "Apache-2.0" +LIC_FILES_CHKSUM = "file://contrib/android/ext2simg.c;beginline=1;endline=14;md5=fc8a826484cc82548adeaad2579adfca" + +DEPENDS = "e2fsprogs-native android-tools-native" + +SRC_URI = "git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git;branch=master;protocol=https" +SRCREV = "ece89fac4603e400155b7bbf6326284f8511bca9" + +inherit native + +do_compile() { + bbnote "Compiling ext2simg.c with native toolchain" + # Source directory for ext2simg.c in the e2fsprogs tree + SRC_EXT2SIMG="${S}/contrib/android" + + INCLUDES="-I${S}/lib -I${B}/lib -I${SRC_EXT2SIMG}/lib" + + # STAGING_LIBDIR_NATIVE: where android-tools-native installed libsparse + LIBS="-L${B}/lib -L${RECIPE_SYSROOT_NATIVE}${libdir} -L${RECIPE_SYSROOT_NATIVE}${libdir}/android -lsparse -lext2fs -lz -lcom_err" + + ${CC} ${CFLAGS} ${LDFLAGS} -o "${B}/ext2simg_android" "${SRC_EXT2SIMG}/ext2simg.c" ${INCLUDES} ${LIBS} + bbnote "ext2simg_android compilation finished" +} + +do_install() { + install -d ${D}${bindir} + install -m 0755 ${B}/ext2simg_android ${D}${bindir}/ext2simg_android +} +
Signature · 1 line
2.43.0
PATCH v3 2/3 android-tools: export libsparse/libbase/liblog headers and libs to sysroot
In reply to AshishKumar MishraExport libsparse, libbase, and liblog headers and libraries to the sysroot. Currently, these are often internal to the build.
Reorganise header installation to ${includedir}/sparse to match the expected include paths for e2fsprogs contrib tools.
--- .../android-tools/android-tools_%.bbappend | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend
meta-oe/recipes-devtools/android-tools/android-tools_%.bbappend new file +54 -0
@@ -0,0 +1,54 @@ +FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:" + +# e2fsprogs expecting headers in sparse/ subdirectory +do_install:append:class-native() { + # to resolve the "Multiple shlib providers" conflict. + rm -f ${D}${libdir}/android/libsparse.so* + rm -f ${D}${libdir}/android/libbase.so* + rm -f ${D}${libdir}/android/liblog.so* + + if [ -d "${S}/system/core/libsparse/include/sparse" ]; then + install -d ${D}${includedir}/sparse + cp -r ${S}/system/core/libsparse/include/sparse/* ${D}${includedir}/sparse/ + else + bbfatal "Sparse headers not found in ${S}/system/core/libsparse/include/sparse" + fi + + install -d ${D}${libdir} + for lib in libsparse libbase liblog; do + if [ -f "${S}/debian/out/system/${lib}.so" ]; then + bbnote "Installing ${lib} from ${S}/debian/out/system/${lib}.so" + install -m 0755 ${S}/debian/out/system/${lib}.so ${D}${libdir}/${lib}.so.0 + ln -sf ${lib}.so.0 ${D}${libdir}/${lib}.so + elif [ -f "${S}/debian/out/system/${lib}.so.0" ]; then + bbnote "Installing ${lib} from ${S}/debian/out/system/${lib}.so.0" + install -m 0755 ${S}/debian/out/system/${lib}.so.0 ${D}${libdir}/${lib}.so.0 + ln -sf ${lib}.so.0 ${D}${libdir}/${lib}.so + else + bberror "${lib} not found in ${S}/debian/out/system" + fi + done + + install -d ${D}${libdir}/android + for lib in libsparse libbase liblog; do + if [ -f "${D}${libdir}/${lib}.so.0" ]; then + ln -sf ../${lib}.so.0 ${D}${libdir}/android/${lib}.so.0 + ln -sf ../${lib}.so ${D}${libdir}/android/${lib}.so + fi + done +} + +FILES:${PN}-dev += " \ + ${includedir}/sparse \ + ${libdir}/lib*.so \ + ${libdir}/android/lib*.so \ +" + +FILES:${PN} += " \ + ${libdir}/lib*.so.* \ + ${libdir}/android/lib*.so.* \ +" + +SYSROOT_DIRS:append:class-native = " ${includedir} ${libdir}" + +DEPENDS:append:class-native = " googletest-native"
Signature · 1 line
2.43.0
PATCH v3 3/3 image_types_sparse: switch ext* conversion to ext2simg_android
In reply to AshishKumar MishraWe have RFC discussion at
https://lists.openembedded.org/g/openembedded-devel/message/124499
https://lists.openembedded.org/g/openembedded-devel/message/125497
Move ext* sparse image generation from 'img2simg' to ' ext2simg_android'
--- meta-oe/classes/image_types_sparse.bbclass | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
meta-oe/classes/image_types_sparse.bbclass +13 -3
@@ -13,11 +13,21 @@ DELETE_RAWIMAGE_AFTER_SPARSE_CMD ??= "0" CONVERSION_CMD:sparse = " \ truncate --no-create --size=%${SPARSE_BLOCK_SIZE} "${IMAGE_NAME}.${type}"; \ - img2simg -s "${IMAGE_NAME}.${type}" "${IMAGE_NAME}.${type}.sparse" ${SPARSE_BLOCK_SIZE}; \ + case '${type}' in \ + ext*) \ + bbnote 'Running e2fsprogs-derived ext2simg_android..' ; \ + ext2simg_android '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.${type}.simg' || bberror 'ext2simg_android failed' \ + ;; \ + *) \ + bbnote 'Generating sparse image for non-ext filesystem...'; \ + img2simg -s '${IMAGE_NAME}.${type}' '${IMAGE_NAME}.${type}.sparse' ${SPARSE_BLOCK_SIZE}; \ + ;; \ + esac; \ if [ "${DELETE_RAWIMAGE_AFTER_SPARSE_CMD}" = "1" ]; then \ rm -f ${IMAGE_NAME}.${type};\ - bbwarn "Raw file ${IMAGE_NAME}.${type} removed" ;\ + bbnote "Raw file ${IMAGE_NAME}.${type} removed" ;\ fi;\ " -CONVERSION_DEPENDS_sparse = "android-tools-native" +CONVERSION_DEPENDS_sparse = "android-tools-native e2fsprogs-ext4sparse-native" +do_image_ext4[depends] += "e2fsprogs-ext4sparse-native:do_populate_sysroot"
Signature · 1 line
2.43.0
Have update in the RFC-2 thread in parallel https://lists.openembedded.org/g/openembedded-devel/message/124499
Hi Team ,
Can team please take this in consideration in master branch
If team have any specific feedback , i can quickly work and submit accordingly
Thanks ,
Ashish